Image resolution multiplier

ABSTRACT

A system for generating high resolution image data using a low resolution image sensor is provided. The system includes an image sensor array and an image size system determining a pixel resolution area. An image target system determines a desired pixel resolution area, and an image sensor array placement system determines two or more locations for an image sensor array based on the pixel resolution area and the desired pixel resolution area. An image composition system receives first image data of a target from the image sensor array at a first location and second image data of the target from the image sensor array at a second location and combines the first image data and the second image data to form composite image data having an effective pixel resolution area less than the pixel resolution area.

FIELD OF THE INVENTION

The present invention relates to image data generation, and morespecifically to movement of a pixel array to locations that are lessthan the pitch between pixels so as to increase image resolution.

BACKGROUND

Image data resolution improvement systems are known. Such systemstypically use magnifiers to increase the effective resolution anddecrease the effective pixel resolution area. Such magnification systemssuffer from the drawback that magnification decreases light intensity,which requires longer periods of time for image data to be generated.Likewise, magnification ratios are typically fixed and dependent onavailable optical lenses.

SUMMARY OF THE INVENTION

In accordance with the present invention, an image resolution multiplieris provided that overcomes known problems with image resolutionmultipliers.

In particular, an image resolution multiplier is provided that allows apixel array to be moved distances that are less than the distancebetween two adjacent pixels, so as to provide for improved pixelresolution in the combined image without any need for an increase inlighting requirements.

In accordance with an exemplary embodiment of the present invention, asystem generating high resolution image data using a low resolutionimage sensor is provided. The system includes an image sensor and animage size system determining a pixel resolution area. An image targetsystem determines a desired pixel resolution area, and an image sensorarray placement system determines two or more locations for an imagesensor array based on the pixel resolution area and the desired pixelresolution area. An image composition system receives first image dataof a target from the image sensor array at a first location and secondimage data of the target from the image sensor array at a secondlocation and combines the first image data and the second image data toform composite image data having an effective pixel resolution area lessthan the pixel resolution area.

The present invention provides many important technical advantages. Oneimportant technical advantage of the present invention is a system forimproving image data resolution that does not require opticalmagnification. Other important technical advantage of the presentinvention is a system for generating high resolution image data, such asa set of 4000×4000 pixel image data, using a low resolution image sensorarray, such as a 1000×1000 pixel sensor.

Those skilled in the art will further appreciate the advantages andsuperior features of the invention together with other important aspectsthereof on reading the detailed description that follows in conjunctionwith the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of a system for generating high resolution imagedata using a low resolution image sensor in accordance with an exemplaryembodiment of the present invention;

FIG. 2 is a diagram of a system for image sensor placement in accordancewith an exemplary embodiment of the present invention;

FIG. 3 is a diagram of a system for combining sets of image data to forma composite image in accordance with an exemplary embodiment of thepresent invention; and

FIG. 4 is a flow chart of a method for generating composite image datain accordance with an exemplary embodiment of the present invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

In the description that follows, like parts are marked throughout thespecification and drawings with the same reference numerals,respectively. The drawing figures might not be to scale, and certaincomponents can be shown in generalized or schematic form and identifiedby commercial designations in the interest of clarity and conciseness.

FIG. 1 is a diagram of a system 100 for generating high resolution imagedata using a low resolution image sensor in accordance with an exemplaryembodiment of the present invention. System 100 allows an image sensorwith a predetermined pixel resolution area to be moved so as to decreasethe effective pixel resolution area and correspondingly increase imagedata resolution.

System 100 includes image target area 102 and image sensor array system104. Image sensor array system 104 can be a CMOS sensor or othersuitable sensor array that generates image data of an item in imagetarget area 102. Image sensor array system 104 or image target area 102can also include stepper motors or other devices that can be used tochange the location of an item in image target area 102 relative toimage sensor array system 104. In one exemplary embodiment, image sensorarray system 104 and image target area 102 can be moved relative to eachother such that the displacement of the image sensor array system 104relative to the item being inspected in image target area 102 is afraction of the distance between two pixels of image sensor array system104. Likewise, image sensor array system 104 or image target area 102can be moved along a Z-axis direction such as to increase or decreasethe distance between image sensor or array system 104 and image targetarea 102. In this manner, 2-dimensional image data arrays can begenerated along a third orthogonal axis, such as to generatethree-dimensional data of an inspection item.

Image sensor enhancer system 106 can be implemented in hardware,software, or a suitable combination of hardware and software, and can beone or more software systems operating on a general purpose processingplatform. As used herein, a hardware system can include discretesemiconductor devices, an application-specific integrated circuit, afield programmable gate array, a general purpose processing platform, orother suitable devices. A software system can include one or moreobjects, agents, threads, lines of code, subroutines, separate softwareapplications, user-readable (source) code, machine-readable (object)code, two or more lines of code in two or more corresponding softwareapplications, databases, or other suitable software architectures. Inone exemplary embodiment, a software system can include one or morelines of code in a general purpose software application, such as anoperating system, and one or more lines of code in a specific purposesoftware application.

Image sensor enhancer system 106 is coupled to image target area 102 andimage sensor array system 104. As used herein, the term “couple” and itscognate terms, such as “couples” and “coupled,” can include a physicalconnection (such as a copper conductor), a virtual connection (such asthrough randomly assigned memory locations of a data memory device), alogical connection (such as through logical gates of a semiconductingdevice), other suitable connections, or a suitable combination of suchconnections. In one exemplary embodiment, systems and components arecoupled to other systems and components through intervening systems andcomponents, such as through an operating system. Communications mediacan be a local area network, a wide area network, a public network suchas the Internet, the public switched telephone network, a wirelessnetwork, a fiber optic network, other suitable media, or a suitablecombination of such media.

Image sensor enhancer system 106 includes image size system 108, imagetarget system 110, image sensor array placement system 112, imagecomposition system 116, image inspection system 118, and resolutionadjustment system 120, each of which can be implemented in hardware,software, or a suitable combination of hardware and software and whichcan be one or more software systems operating on a general purposeprocessing platform. Image size system 108 determines the pixelresolution area of an image sensor array. In one exemplary embodiment,image size system 108 can receive the size of target area 102, a numberof pixels in a pixel array, or other suitable data that can allow imagesize system 108 to determine the pixel resolution area.

Image target system 110 receives inspection item data and determines adesired pixel resolution area. In one exemplary embodiment, theinspection item data can include the size of one or more features on aninspection item, such as a width, a length, a circumference, a height,or other suitable data, as well as data identifying the size ofcharacters, markings, components, or other features on the inspectionitem, so as to allow a desired or required pixel resolution area to bedetermined.

Image sensor array placement system 112 receives image pixel area datafrom image size system 108 and required pixel resolution area data fromimage target system 110 and generates image sensor array placement data.In one exemplary embodiment, image sensor array placement system 112 caninclude a plurality of locations for an image sensor array based onpredetermined desired pixel resolution areas and known pixel resolutionarea requirements for inspection items and configuration parameters ofimage sensor array system 104. In another exemplary embodiment, imagesensor array placement system 112 can determine the amount of movementrequired and number of image data sets required in order to form acomposite image having the required effective or desired pixelresolution area.

For example, it may be determined that the effective or desired pixelresolution area would require the image sensor array to be moved to twointermediate locations between the locations of pixels in the pixelarray. For example, if the distance between pixels (or pixel pitch) is 1millimeter and a resolution of at least 0.5 millimeters in the x-axisdirection is required, then the following process might be used:

-   -   Generation of image data at a first location,    -   Movement of the image sensor array from the first location to a        second location along the x-axis that is one third of the pixel        pitch away from the initial location (such as to provide greater        than 0.5 millimeter accuracy),    -   Generation of image data at the second location,    -   Movement to a third location two-thirds of the pixel pitch        distance between pixels, and    -   Generation of image data.        Likewise, similar movement may be required in an orthogonal        direction, such as where the first movement is along an X-axis        and the second movement is along the Y-axis. Other suitable        coordinate systems can be used such as Cartesian coordinate        systems where the image sensor array is rotated, or single axis        coordinate movement where the image array data of interest only        changes along a single axis.

Image composition system 116 receives sets of image data from imagesensor array placement system 112 and data from image size system 108and image target system 110 and assembles the sets of image data into acomposite image. In one exemplary embodiment, image composition system116 can create a composite image based on the known resolutioncapabilities of each pixel, such as where edge effects or other effectsmay cause the image data generated at a second location to be givenpreference over values of brightness or other values generated at afirst location. For example, image sensor array system 104 can generatechrominance, luminance, or other suitable data, and image compositionsystem 116 can determine whether to choose the chrominance, luminance orother data at a point over the chrominance, luminance or other data atan associated point, whether to combine the two chrominance, luminanceor other data values and take an average, whether to use some otherarithmetic relationship to determine a chrominance, luminance, or otherdata value for that point, or to otherwise suitably combine sets ofimage data. In another exemplary embodiment, where image sensor arraysystem 104 is rotated, the degree of pixel resolution increase will begreater near the center of the axis of rotation and less around theperiphery of the image sensor array. Image composition system 116 cangenerate suitable compensation factors based upon the axis of movement.Likewise, if sets of image data are generated at locations along aZ-axis, where each set is generated by moving image sensor array system104 in X and Y-axis directions, image composition system 116 can be usedto generate three-dimensional mapping, such as to estimate the size of asurface defect or other features.

Image inspection system 118 receives image data from image compositionsystem 116 and generates image inspection data. In one exemplaryembodiment, image inspection system 118 can determine whether desiredinspection features are present in the data received from imagecomposition system 116, such as indexing features that are used to setboundaries for an area to be inspected, marking data, or other suitabledata.

Resolution adjustment system 120 receives resolution adjustment datafrom image inspection system 118 and provides resolution adjustment datato image target system 110. In one exemplary embodiment, resolutionadjustment system 120 can be used to generate new sets of inspectiondata when the effective pixel resolution area data provided by imagetarget system 110 did not provide a sufficient level of image resolutionto allow image inspection system 118 to perform an image inspection.Likewise, resolution adjustment system 120 can change resolution basedon the type of device being inspected, the area of the device beinginspected, or other suitable information.

In operation, image sensor enhancer system 106 allows an image sensorarray system 104 to be moved in such a manner so as to improve theeffective resolution by decreasing the effective pixel resolution area.In this manner, an image sensor array that would normally provide apredetermined effective pixel resolution area can be used to decreasethe effective pixel resolution area so as to provide more data for aninspection item. For example, if a feature on an inspection item has asize of “N,” and the maximum resolution based on location of pixels inimage sensor array system 104 is 10N, it is possible to combine a numberof sets of image data from locations between each pixel location bymoving image sensor array system 104 in X and Y directions at fractionsof the pixel pitch, such as 0.1N in the given example. In this manner,suitable image resolution can be obtained, allowing suitable data to beobtained.

FIG. 2 is a diagram of a system 200 for image sensor placement inaccordance with an exemplary embodiment of the present invention. System200 includes image sensor array placement system 112 and X-axisplacement system 202, Y-axis placement system 204, Z-axis placementsystem 206, and Cartesian placement system 208, each of which can beimplemented in hardware, software, or a suitable combination of hardwareand software and in which one or more software systems operating on ageneral purpose processing platform.

X-axis placement system 202 generates movement data for image sensorarray system 104, image target area 102, or other suitable devices orstructures so as to allow an image sensor array to be moved apredetermined amount or within degrees or fractions of the pitch betweenpixels of the image sensor array. X-axis placement system can be used togenerate X-axis movement, can be used in conjunction with Y-axisplacement system 204 to move the image sensor array system 104 to a newlocation at a new X and Y coordinate, can be used in conjunction withZ-axis placement system 206 to change the amount of X-axis movementbased on a Z-axis location, can be used in conjunction with Cartesianplacement system 208 to provide effective X and Y coordinate data formovement in a Cartesian coordinate system, or can be used in othersuitable manners.

Y-axis placement system 204 generates movement data for image sensorarray system 104, image target area 102, or other suitable devices orstructures so as to allow an image sensor array to be moved apredetermined amount or within degrees or fractions of the pitch betweenpixels of the image sensor array. Y-axis placement system 204 can beused to generate Y-axis movement, can be used in conjunction with X-axisplacement system 202 to move the image sensor array system 104 to a newlocation at a new X and Y coordinate, can be used in conjunction withZ-axis placement system 206 to change the amount of Y-axis movementbased on a Z-axis location, can be used in conjunction with Cartesianplacement system 208 to provide effective X and Y coordinate data formovement in a Cartesian coordinate system, or can be used in othersuitable manners.

Z-axis placement system 206 generates movement data for image sensorarray system 104, image target area 102, or other suitable devices orstructures so as to allow an image sensor array to be moved apredetermined amount based on the location of an inspection item along aZ-axis. Z-axis placement system 206 can be used to generate Z-axismovement, can be used in conjunction with X-axis placement system 202and Y-axis placement system 204 to move the image sensor array system104 to a new location at a new X and Y coordinate based on Z-axiscoordinates, can be used in conjunction with Cartesian placement system208 to provide Cartesian coordinate system movement as a function ofZ-axis locations, or can be used in other suitable manners.

Cartesian placement system 208 generates movement data for image sensorarray system 104, image target area 102, or other suitable devices orstructures so as to allow an image sensor array to be moved apredetermined amount or within degrees or fractions of the pitch betweenpixels of the image sensor array. Cartesian placement system 208 can beused to movement within a Cartesian coordinate system, can be used inconjunction with Z-axis placement system 206 to change the amount ofCartesian coordinate movement based on a Z-axis location, can be used inconjunction with X-axis placement system 202 and Y-axis placement system204 to generate x and Y coordinate data, or can be used in othersuitable manners.

In operation, system 200 allows a pixel array to be moved in a mannerthat allows image data resolution to be improved by decreasing theeffective pixel resolution area, system 200 further allows the changesin resolution to be controlled based upon inspection itemcharacteristics that vary along an X-axis, a Y-axis, a Z-axis, or withina Cartesian coordinate system, or based on a suitable combination ofsuch coordinate data.

FIG. 3 is a diagram of a system 300 for combining sets of image data toform a composite image in accordance with an exemplary embodiment of thepresent invention. System 300 includes image composition system 116,inspection set definition system 302, image set storage system 304, setcombination system 306, and inspection image system 308, each of whichcan be implemented in hardware, software, or a suitable combination ofhardware and software, and which can be one or more software systemsoperating in general purpose processing platform.

Inspection set definition system 302 receives pixel placement data foruse in defining inspection sets. In one exemplary embodiment, inspectionset definition system 302 can receive pixel resolution area andeffective pixel resolution area and can generate inspection setdefinition data that identifies the number of sets of inspection datathat need to be generated in order to provide the effective pixelresolution area. Likewise, inspection set definition system 302 caninclude pre-determined inspection definition data, such as inspectiondata that varies as a function of location of a device, the type ofdevice being inspected, or other suitable inspection definition data.

Image set storage system 304 receives two or more sets of image data foran inspection item. In one exemplary embodiment, image set storagesystem 304 can receive the sets of image data for the item and storethem so that the image data can be processed at a later time, such asafter the generation of all sets of image data. Likewise, image setstorage system 304 can include buffers that store the data forprocessing that occurs as additional image data for an inspection itemis being generated. Other suitable processes can also be used.

Set combination system 306 receives image set data from image setstorage system 304 and combines the image set data to form a compositeimage. In one exemplary embodiment, set combination system 306 can usepixel characteristic data to determine how to combine the pixel datasets. For example, if the effective pixel resolution is four timessmaller than the actual pixel resolution for the image sensor arraysystem 104 (image target area 102/number of pixels in array), then setcombination system 306 can receive a suitable number of sets of imagedata from the image sensor array system 104 taken at suitable locations,so as to yield an effective pixel resolution that meets or exceeds thedesired effective pixel resolution.

Inspection image system 308 generates an inspection image data set foruse by image inspection system 118. In one exemplary embodiment,inspection image system 308 can store the inspection image data forsubsequent use, for review by an operator, or for other suitablepurposes. Likewise, inspection image system 308 can generate calibrationimages, can compare inspection image data to calibration data, templatedata, or other suitable data to determine whether the set of inspectionimage data falls within an allowable range image data values, or othersuitable processes can be performed.

In operation, system 300 allows multiple sets of image data to becombined to form a set of inspection image data so as to increase theeffective inspection resolution, decrease the effective pixel resolutionarea, and for other suitable purposes. System 300 thus allows an imagesensor array to be used to inspect components where features of thecomponent have a smaller size than the image sensor array couldotherwise detect without magnification.

FIG. 4 is a flow chart of a method 400 for generating composite imagedata in accordance with an exemplary embodiment of the presentinvention. Method 400 begins at 402 where the pixel resolution isdetermined. In one exemplary embodiment, a pixel resolution can bedetermined based upon the size of a pixel sensor array and an imagetarget area, any magnification factors, or other suitable data. Themethod then proceeds to 404.

At 404 it is determined whether the resolution that has been determinedis acceptable. In one exemplary embodiment, the inspection area can be a10 millimeters square area, and the number of pixels used to analyze theinspection area can be a 1000×1000 array, such that the resolution areais a 0.01-millimeter square area. If it is determined that the itembeing inspected has no features that are smaller than the effectiveresolution, the method proceeds to 406 and the resolution is set.Otherwise the method proceeds to 408.

At 408 an adjustment ratio is determined. In one exemplary embodiment,the adjustment ratio can be related to a number of incremental spotsbetween each pixel to which the pixel sensor array must be moved. Inanother exemplary embodiment, the adjustment ratio can be based onmovement of the pixel array to locations relative to the current pixelarray location, such as X-axis and Y-axis movement coordinates,Cartesian movement coordinates, movement coordinates in the X-axis andY-axis direction based on a new Z-axis location, or other suitableadjustment ratios. The method then proceeds to 410.

At 410 the image sensor array positions are set. In one exemplaryembodiment, image sensor array system 104, image target area 102, orother suitable structures can be moved relative to each other so as toallow an accurate location of the image sensor array to be performed.The image sensor array positions can be set based on calibration data,data from calibrated Stepper motors or other mechanical devices, orother suitable sensor array position data. The method then proceeds to412.

At 412 image data is generated at a first location. In one exemplaryembodiment, this first location can be the default location forgeneration of image data. Likewise, the image data can be generated at afirst location that is not the default image sensor array systemlocation relative to an image target area, or other suitable locationscan be used. The method then proceeds to 416.

At 416 it is determined if it is necessary to generate image data atadditional locations. If image data from additional locations isrequired, the method proceeds to 418 where the pixel sensor array ismoved to the next location. For example, an inspection set definitionsystem 302 or image sensor array placement system 112 can be consultedto determine the location based on the current pixel image sensor arraylocation. The method then proceeds to 420 where the image data set isgenerated at the new location. The image data set can be stored, can becompared to or combined with the first image data set, or the suitableprocesses can be used. The method then returns to 416.

If it is determined at 416 that there are no additional locations atwhich image data should be generated, then the method proceeds to 422where the sets of data are combined. As previously described, if setshave been combined as they are being generated, the final set of imagedata can be combined at 422. Likewise, other suitable processes can beused, such as combination of sets of image data that utilizemultiple-pixel averaging or other mathematical relationships for threeor more pixels. The method then proceeds to 424.

At 424 the combined data is analyzed to determine whether inspectionprocesses can be performed on the combined data. In one exemplaryembodiment, identification of one or more pre-determined features of aninspection item in a set of image data can be determined, or othersuitable processes can be performed to determine whether the data issuitable for analysis. The method then proceeds to 426.

At 426 it is determined whether acceptable resolution has been provided.If acceptable resolution has not been provided, the method returns to408 where an adjustment ratio is determined. Otherwise, the methodproceeds to 428 and the combined sets are used to perform inspectionprocesses, are stored, or other suitable processes are performed.

In operation, method 400 allows an image sensor array to be moved to asub-pitch location so as to increase the effective image resolutionusing an image sensor array with a lower resolution. Method 400 thusallows an image sensor array to be used in conjunction with otherprocesses that may not require high resolution, to be used to alterresolution based on variations and a Z-axis or other direction of animage test component, or for other suitable purposes.

Although exemplary embodiments of a system and method of the presentinvention have been described in detail herein, those skilled in the artwill also recognize that various substitutions and modifications can bemade to the systems and methods without departing from the scope andspirit of the appended claims.

1. A system generating high resolution image data using a low resolutionimage sensor comprising: an image sensor array having a predeterminedimage resolution; an image size system determining a pixel resolutionarea; an image target system determining a desired pixel resolutionarea; an image sensor array placement system determining two or morelocations for an image sensor array based on the pixel resolution areaand the desired pixel resolution area; and an image composition systemreceiving first image data of a target from the image sensor array at afirst location and second image data of the target from the image sensorarray at a second location and combining the first image data and thesecond image data to form composite image data having an effective pixelresolution area less than the pixel resolution area.
 2. The system ofclaim 1 wherein the image sensor array placement system furthercomprises an x-axis placement system determining two or more locationsfor the image sensor array along an x-axis relative to the target. 3.The system of claim 1 wherein the image sensor array placement systemfurther comprises an x-axis placement system determining two or morelocations for the image sensor array along an x-axis relative to thetarget based on the desired pixel resolution area.
 4. The system ofclaim 1 wherein the image sensor array placement system furthercomprises an y-axis placement system determining two or more locationsfor the image sensor array along a y-axis relative to the target.
 5. Thesystem of claim 1 wherein the image sensor array placement systemfurther comprises an y-axis placement system determining two or morelocations for the image sensor array along a y-axis relative to thetarget based on the desired pixel resolution area.
 6. The system ofclaim 1 wherein the image sensor array placement system furthercomprises a z-axis placement system determining two or more locationsfor the image sensor array along a z-axis relative to the target.
 7. Thesystem of claim 1 wherein the image sensor array placement systemfurther comprises a z-axis placement system determining two or morelocations for the image sensor array along a z-axis relative to thetarget based on the desired resolution along the z-axis.
 8. The systemof claim 1 wherein the image sensor array placement system furthercomprises a z-axis placement system determining two or more locationsfor the image sensor array along a z-axis relative to the target basedon the desired resolution along z-axis, wherein the desired resolutionalong z-axis varies as a function of z-axis location.
 9. The system ofclaim 1 wherein the image sensor array placement system furthercomprises a Cartesian coordinate placement system determining two ormore locations for the image sensor array at two or more Cartesiancoordinates relative to the target.
 10. The system of claim 1 whereinthe image composition system further comprises a weighting factor systemthat applies a weighting factor when combining the first image data andthe second image data to form the composite image data.
 11. A method forgenerating high resolution image data using a low resolution imagesensor comprising: determining a pixel resolution area; determining adesired pixel resolution area; determining two or more locations for animage sensor array based on the pixel resolution area and the desiredpixel resolution area; and receiving first image data of a target fromthe image sensor array at a first location and second image data of thetarget from the image sensor array at a second location and combiningthe first image data and the second image data to form composite imagedata having an effective pixel resolution area less than the pixelresolution area.
 12. The method of claim 11 wherein determining two ormore locations for the image sensor array comprises determining two ormore locations for the image sensor array along an x-axis relative tothe target.
 13. The method of claim 11 wherein determining two or morelocations for the image sensor array comprises determining two or morelocations for the image sensor array along an x-axis relative to thetarget based on the desired pixel resolution area.
 14. The method ofclaim 11 wherein determining two or more locations for the image sensorarray comprises determining two or more locations for the image sensorarray along a z-axis relative to the target based on the desired z-axisresolution as a function of z-axis location.
 15. The method of claim 11wherein combining the first image data and the second image data to formcomposite image data having an effective pixel resolution area less thanthe pixel resolution area comprises applying a weighting factor whencombining the first image data and the second image data to form thecomposite image data.
 16. The method of claim 11 wherein combining thefirst image data and the second image data to form composite image datahaving an effective pixel resolution area less than the pixel resolutionarea comprises applying a weighting factor based on the first locationand the second location.
 17. An image composition system for generatinghigh resolution image data using a low resolution image sensorcomprising: a first target image data system receiving image data of atarget from an image sensor array at a first location; a second targetimage data system receiving image data of the target from the imagesensor array at a second location; and and a combination systemcombining the first image data and the second image data to formcomposite image data having an effective pixel resolution area less thanthe pixel resolution area.
 18. The system of claim 17 further comprisingan image sensor array location system determining the first location andthe second location based on desired pixel resolution area.
 19. Thesystem of claim 17 further comprising an image sensor array locationsystem determining the first location and the second location based ondesired pixel resolution area.
 20. The system of claim 17 furthercomprising an image sensor array location system determining the firstlocation and the second location based on effective pixel resolutionarea.